System for annotating an object in a video

ABSTRACT

A system providing a user interface to annotate different items in a media production system such as in a digital non-linear post production system. Parts of the production, such as clips, frames and layers, that have an associated annotation are provided with a visual annotation marker. The annotation marker can use shape, color or animation to convey source, urgency, status or other information. Annotations can be text, freehand drawing, audio, or other. Annotations can be automatically generated. Annotations can be compiled into records, searched and transferred. A state of an application program can be stored and transferred to a remote system. The remote system attempts to recreate the original state of the application program. If the remote system is unable to do so, an image of the state of the application program is obtained, instead. Assignment of control to various functions of an application program is achieved by associating a function (i.e., modifying a parameter) with a user control at a remote location.

CLAIM OF PRIORITY

This application claims priority from the following co-pending U.S.Provisional Patent Applications:

1. Application Ser. No. 60/271,376; filed Feb. 22, 2001, entitled “ASystem and Method for Editing” (client docket 50P4410); and

2. Application Ser. No. 60/284,660; filed Apr. 17, 2001, entitled“Advanced System and Method for Editing” (client docket 50R4639).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to the following co-pending U.S. patentapplications:

1. application Ser. No. 09/665,721; filed Sep. 18, 2000, entitled“System And Method For Optimizing The Processing Of Images” (clientdocket 50N3647);

2. application Ser. No. 09/691,795; filed Oct. 18, 2000, “System AndMethod For Increasing Performance When Compositing Images” (clientdocket 50N3649);

3. [Ser. No. and filing date TBA] entitled “User Interface forGenerating Parameter Values in Media Presentations Based on SelectedPresentation Instances” (attorney docket 20699-45);

4. [Ser. No. and filing date TBA] entitled “Media Production SystemUsing Flowgraph Representation of Operations” (attorney docket20699-47); and,

5. [Ser. No. and filing date TBA] entitled “User Interface for a DigitalProduction System Including Multiple Viewing and Navigating” (attorneydocket 20699-48);

BACKGROUND OF THE INVENTION

Today, digital systems are used to represent and process virtually alltypes of media in commercial, entertainment, educational and otherfields. For example, visual media such as photographs, movies, video andanimations are largely handled in the digital domain. Similarly, audiomedia such as speech, music, sound effects, etc., are also recorded,edited, produced, copied and transferred in the digital domain.

The production systems used to produce and manipulate digital mediaperform many other functions such as adding content and auxiliaryinformation to the media, transferring portions of the media for reviewand approval, storing and indexing the media, etc. There are manypossible operations on even a small media portion, such as on one ofthousands of frames in a movie. These operations are often repeated manytimes by trial-and-error, and then extrapolated to larger segments ofthe production. There may be many hundreds or thousands of operationsperformed over weeks of creation. This is further compounded when manypeople are involved in creating, reviewing, or approving the productionat steps along the way. Often, some of the people involved in theproduction are not technically inclined and, yet, must work in thehighly technical and specialized environment provided by a sophisticatedproduction system. To make the situation even more difficult, theproduction team members may be located in geographically remote placesthus making communication and collaboration extremely difficult.

Thus, it is desirable to provide production system features to improveupon one or more shortcomings in the prior art.

BRIEF SUMMARY OF THE INVENTION

The invention provides a user interface to annotate different items in amedia production system such as in a digital non-linear post productionsystem. Parts of the production, such as clips, frames and layers, thathave an associated annotation are provided with a visual annotationmarker. The annotation marker can use shape, color or animation toconvey source, urgency, status or other information. Annotations can betext, freehand drawing, audio, or other. Annotations can beautomatically generated. Annotations can be compiled into records,searched and transferred. A state of an application program can bestored and transferred to a remote system. The remote system attempts torecreate the original state of the application program. If the remotesystem is unable to do so, an image of the state of the applicationprogram is obtained, instead. Assignment of control to various functionsof an application program is achieved by associating a function (i.e.,modifying a parameter) with a user control at a remote location.

In one embodiment the invention provides a method for annotating an itemin a user interface of a media production system. The method includesaccepting signals from a user input device to select a part of aproduction being processed by the media production system; creatingannotation information; and storing the annotation information inassociation with the selected part of the production.

In another embodiment the invention provides a method for conveying thestate of an application program to a remote location. The methodincludes determining to capture the current state of operation of theapplication program; saving one or more settings of a currentenvironment of the application program as a recorded environment; andtransferring the recorded environment to a remote location.

In another embodiment the invention provides a method for recreating thestate of an application program at a process executing at a locationremote from the application program, wherein a recorded environmentincludes settings for recreating the state of the application program.The method including receiving, over a digital network, the recordedenvironment; executing another instance of the application program atthe location remote from the application program; and using the instanceand the recorded environment to recreate the state of the applicationprogram at the location remote from the application program.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates the use of annotation markers on clips and frames;

FIG. 1B shows a flowgraph panel with an annotated node;

FIG. 1C illustrates freehand drawing to annotate a frame;

FIG. 1D shows a persistent text annotation box;

FIG. 1E shows the persistent text annotation box of FIG. 1D in adifferent panel;

FIG. 2 illustrates use of a recorded environment;

FIG. 3 illustrates the remotely assignable control features of thepresent invention; and

FIG. 4 illustrates drag-and-drop assignment of user interface mechanismsat the primary user's computer to control surfaces at a remote user'scomputer.

DETAILED DESCRIPTION OF THE INVENTION

For purposes of this application, any system that is used in associationwith producing a media production, including systems that provide one ormore features for displaying, reviewing, modifying, annotating,transferring, editing, storing, etc., media productions is called a“production system.” Details of a preferred embodiment production systemare discussed in co-pending U.S. patent applications SYSTEM AND METHODFOR OPTIMIZING THE PROCESSING OF IMAGES and SYSTEM AND METHOD FORINCREASING PERFORMANCE WHEN COMPOSITING IMAGES, referenced above.

A preferred embodiment system is referred to as “Socratto” which isdeveloped and marketed by VFX, a division of Sony Corporation. Althoughaspects of the present invention are described in connection with theSocratto system, it should be apparent that the invention is suitablefor use with many different production systems having different ways topresent information and to accept user inputs. Also, although theinvention is described herein primarily with reference to film or videoproduction systems, the invention is applicable to other types of mediaproduction systems such as computer-aided design, audio, animation,modeling, etc. Various aspects of the invention are applicable to anytype of application program or functional system.

The present invention provides extensive annotation features for digitalmedia production systems. Annotations can be image, text or audioinformation associated with parts of a media production such as imagelayers, frames, clips, reels, files, movies or other parts. Groups ofparts can also have associated annotations. Such annotations are usefulin collaborative situations where a note or instruction regarding aspecific part of the production is sent to an operator, artist, etc. Forexample, the instruction “make clip #1839 look more like this” can beassociated with clip #200. Similarly, clip #1839 can have an associatednote such as “not finished—see clip #200 note”. Many other types ofinstructions are possible. Associating instructions with parts of aproduction reduces the need to be in the same physical location to“point” to different parts. Also, the instructions, notes, comments andother annotations can be more easily transferred, catalogued, recalled,compiled and managed.

FIG. 1A illustrates the use of annotation markers on clips and frames.

In FIG. 1A, screen display 100 of the Socratto system includes top panelarea 102, called a “reel” panel, or display. Top panel area 102 is usedto display sequences of frames of visual media (e.g., for a movie)called “clips.” Clips are shown as, e.g., clips 104, 106, 108, 110, 112and 114. Each of four rows of top panel area 102 can be used to displayclips. Each row of clips is referred to as a “reel”. In FIG. 1A,examples of reels are 116, 118, 120 and 122. Note that reel 116 has asingle displayed clip, clip 104. Clip 104 extends beyond the displayarea of screen display 100. There may be more clips associated with reel116 located to the right or left of the displayable area. These clipsremain conceptually associated with reel 116. Reel 118 has threedisplayed clips, namely clips 106, 108 and 110. Reel 120 does not haveany displayed clips, but may have non-displayed clips and otherinformation associated with it. Reel 122 includes displayed clips 112and 114. Reels are organized into scenes. Scenes are collectivelyorganized into a project. Projects, in turn, are associated with a finalproduction. In the present example, the production is a movie.

As is known in the art, each reel can have transport, or playback,controls for playing the segments of the production represented by eachindividual clip. Also, navigation controls can be provided to moveamong, and selectively display and playback, clips in the reels; add,delete and move clips, etc. Other controls can provide for performingoperations on clips, parts of clips, groups of clips, etc. Operationscan include image property adjustments, image transitions, etc. Specificdetails on the general operation of a production system are included inco-pending U.S. patent applications referenced above. Such controls canbe of various types and can be located anywhere in display screen 100,or on additional display screens. Controls can be external physicalsurfaces and devices, virtual mechanisms, etc.

Annotation markers are shown at 140, 142, 144, 146, 148, 150, 152 and154. The annotation markers indicate that there is annotationinformation associated with a frame, clip, clip segment, reel, projector other part of a production. The annotation marker position andgraphic image provide convenient, easily identified, information to auser of the production system without unduly cluttering the display.

In FIG. 1A, annotation markers 140 and 150 are positioned at the middleof a reel's row at the left edge of the screen. Annotation markers 140and 150 are half-circle orange dots to indicate that annotationinformation is associated with reels 116 and 120, respectively.

Annotation marker 146 is a complete dot at the upper-left corner of aframe. This type of single-frame annotation marker tells the user thatthe annotation is intended to relate to the frame underlying annotationmarker 146 only, or that the associated annotation may be most relevantto the underlying frame. Annotation marker 142 is a dot with aforward-pointing arrow in the upper-left corner of a frame of clip 104.This type of forward-pointing annotation marker indicates that theassociated annotation is pertinent to this frame and onward through theclip. As with all annotation markers, there is no need for strictadherence to the predetermined meanings of the annotation markers. Inother words, using a forward-pointing annotation marker where asingle-frame annotation marker would be more appropriate does not affectthe operation of the annotation operation. The annotation meanings andvarious indications are made available for use, as desired. Groups, orteams, of users will typically develop their own styles and new ways ofusing annotation tools. To this end, other embodiments of the inventioncan allow customizable annotation mechanisms, such as user-created iconsor images for annotation markers.

Typically, an annotation's relevance ends when another annotation markeris encountered or at the end of the clip in which the annotation lies.For example, the annotation associated with annotation marker 144 wouldtypically end at the frame before annotation marker 146. Annotationmarker 148 pertains to the entirety of clip 108. Alternatively,“bookend” type annotation markers can be used to indicate the start andend of clip segments (or other production parts). Such an approach isshown with annotation markers 152 and 154 which are forward-pointing andbackward-pointing annotation markers, respectively. Thus, the annotationassociated with these markers is most relevant to the segment of clip114 starting with the frame underlying forward-pointing annotationmarker 152 and ending with the frame underlying annotation marker 154.

Annotation markers can be of different colors, shapes and animations.Colors can be used to identify a person, group, company, stage ofproduction, etc., that generated the annotation. Colors can also be usedto indicate the nature of the annotation, urgency, etc. For example,low-priority annotations can be represented by an orange annotationindicator while urgent, important, or critical annotation markers canuse red. Colors can be changed automatically to indicate aging. A redmarker can be made a deeper shade of red as time goes by. Animatedannotation markers can also be used to convey information about theassociated information. For example, blinking can be used to indicateurgency.

Annotation markers can be customized. Thus, a company that is involvedin a production can annotate parts of the production with a uniqueannotation marker. Such a unique marker can be the company's logo,initials, or other functional or fanciful image. Similarly, individualscan be identified with their initials. One approach provides informationon the annotation author when a pointer is moved over the annotationmarker for a second or two. A pop-up box can show the author, subject,and brief description of the annotation.

Annotation markers can appear in connection with many parts of theproduction and in association with various controls, panels, displays,or other visual objects in the production system. For example, parts ofa production can include multiple layers that make up a single frame. Anannotation marker can appear on top of, or in association with, adisplay of a layer of a frame. Where the layers are red, green and blue(RUB) components, a frame showing just the red component, or layer, istypically displayed on the production system. The layer display can theninclude an annotation marker, similar to the approach of marking frames.Note that controls, panels (e.g., a playback panel), or any otherfeatures of the user interface can be annotated to describe settings,ask questions, etc.

FIG. 1B shows a flowgraph panel with an annotated node. In a preferredembodiment, a flowgraph is used to represent operations graphically with“nodes” which are circles connected to each other by lines. In apreferred embodiment, a flowgraph is used to represent operationsgraphically with “nodes” which are circles connected to each other bylines. Individual nodes can have an annotation marker upon, or adjacentto, the node, such as annotation marker 134 on node 132. Thus,operations can easily be annotated in much the same way as items, orparts, discussed above.

Annotations can be created by various methods. When a user desires tocreate an annotation, the user selects an object to annotate. Typically,such an object (e.g., a frame, clip, layer, flowgraph node, etc.) has agraphic representation on the display screen. In such a case, the usercan indicate the object to be annotated by selecting with a pointer andpressing a predetermined control such as a mouse button or keyboardbutton. Alternatively, a “tool tip” icon can be selected and then theobject indicated. Any means of selecting an object for annotation isacceptable.

Once selected, user input is accepted to define the annotation. Adialogue box can appear to allow the user to type a text message. Sincesome users may be running the production system without a keyboard, a“softkey” keyboard can appear on the screen so that the user can selectletters and other characters to create a text message. When the textmessage is completed, the annotated part is marked according to themethods described above. Text can occupy several pages. A rudimentary,or complex, text editor can be provided to help the user create textannotations.

A preferred embodiment of the invention accepts freehand “drawing” onthe object, itself, as an annotation.

FIG. 1C illustrates freehand drawing to annotate a frame.

In FIG. 1C, display screen 150 shows frame 152. The user has usedfreehand drawing to manipulate a mouse, graphic tablet, trackball, touchscreen, or any type of input device to scrawl the phrase “THIS IS THECOLOR!” on the image and to draw an arrow and circle around a specificcolor. When the user has completed drawing, the drawn annotation issaved and associated with the frame. The frame is marked with anannotation marker. Selection of the annotation marker places the drawninformation over the image so “pointing,” or other indicating in theannotation is preserved. Note that the drawn information can extendbeyond the frame area into other parts of the display screen. This isuseful so that comments and indications can be made without obscuringportions of the image. The drawing does not destroy the visualrepresentation of the object but is a separate overlay onto the objectand the display area. Drawing tools can be provided such as a pencil,paintbrush, eraser, etc., tools.

Both text and drawing are visual annotations. Another type of visualannotation is an image such as a photograph or bitmap. Any type ofdigital image can be associated with a production part, or object, as anannotation. For example, a thumbnail image of a frame can be associatedwith another frame so that the frame becomes an image-annotated frame.The image-annotated frame will have an annotation marker upon it. Whenselected, the associated image is displayed at a position designated bythe user who annotated the frame. Images, text and drawing types ofannotation can be mixed and stored together as a single annotationassociated with an object. In general, when visual annotations aredisplayed they can appear adjacent to, on top of, or overlapping theobjects that they annotate, or other displayed items on the screen.

Another type of annotation includes a link to other information. Forexample, an annotation can launch a web browser to display a web page.Other applications can be launched to provide other information such asvideo playback of a clip, opening another project in another instance ofthe production system software, etc.

Yet another type of annotation is automatically created. For example, a“slate” annotation option can be enabled by the user. This causes theproduction system (or any other software) to add frames to the clip, orother production part, to include information from a database, or otherprogram, utility or process, to add an artists' name, scene name,version number, date, company logo, etc.

One problem with using annotations is that a typical production systemincludes several different types of panels, or views. Often the user isswitching between panels and views in order to perform operations on themedia. Although such panel switching is necessary to provide enoughviewspace to present information in an efficient and orderly manner, itoften means that an item of interest in one panel is not visible inanother. This might require the user to quickly switch back-and-forthbetween two or more panels to be able to achieve an operation.

In some cases, the annotation may include instructions for operations tobe performed in a panel that is different from the panel that isdisplaying the annotation. A preferred embodiment of the inventionallows annotations, objects, controls, and any other displayed item tobe persistent among different panels. In order to make an itempersistent, the user selects a tool and applies it to the item. The toolcan be a “pushpin” icon on a toolbar. After the pushpin is selected,e.g., by “clicking” on the pushpin icon, it is applied to an item, e.g.,by clicking on the item. The item is then persistent even though thepanel changes.

FIG. 1D shows a persistent text annotation box.

In FIG. 1D, screen display 160 includes a frame display 162 andassociated text annotation in text annotation box 164. Pushpin icon 166has been placed onto text annotation box 164 to indicate that the boxhas been made persistent.

FIG. 1E shows persistent text annotation box 164 of FIG. 1D in adifferent panel.

In FIG. 1E, text annotation box 164 remains on the display screen evenwhen panels are switched. In FIG. 1E, the panel type is the reel panel,discussed above, in connection with FIG. 1A. Note that the persistentannotation box remains visible on top of all of the reel clips andframes.

Typically, additional controls and information are provided in eachpanel view. For ease of use such controls and information are not shown.The persistent item can remain on top of any other displayed items or itcan be assigned a display priority so that it is on top of, under, orbetween other displayed items. Priority assignments can be made by theuser. Another approach is to make the persistent item transparent orsemi-transparent so that items below the persistent item are notcompletely obscured.

Thus, by using persistent annotations, an annotation that givesinstructions for the use of different panels, or views, can remainvisible in different panels.

Annotations can be handled like traditional desktop windows and documentdata in that they can be resized, dragged, minimized, closed, copied,stored, etc. This provides another way to move annotations around in thepanel area to aid in user interface control operations.

The invention provides for audio annotation. A voice recording (or anyother audio) can be associated with an object or item. Thevoice-annotated object can use the same annotation marker, or adifferent marker (e.g., a “speaker” icon) that indicates an audioannotation. When the audio annotation marker is activated (e.g., byclicking on the marker) the audio is played back.

There may be many annotations in a given production, or project. Thepresent invention provides navigation, display and organization featuresso that a user can gain more benefit from annotations.

A navigation system allows the user to move from one annotation to thenext. In the reel panel shown in FIG. 1A, if annotation marker 142 isthe “current” marker (i.e., the last one selected) then the user canadvance successively to markers 144 and 146 by, e.g., pressing a hotkey,clicking on a transport icon, etc. From marker 146, clip 104 is scrolledforward to reveal the next annotation marker on the clip, or on the nextclip within the reel. Similarly, the user can navigate backward frommarker to marker.

Another approach is to provide a list of markers in a menu and jump tothe marker (by displaying the frame, clip, or other part associated withthe marker) after the user selects a marker from the menu.

Markers can be filtered so that only certain types of annotation markersare displayed while others are “hidden” or turned off. For example, theuser can select audio markers, text markers, markers from a givenindividual, etc., for filtering. Other rules for filtering, searchingand displaying markers can be employed. For example, one type of objectthat is important in production operations is called a “keyframe.” Akeyframe is usually the point where an operation begins. Thus, it may beuseful for the user to filter out all annotation markers besides thosethat pertain to keyframes.

When annotations are hidden, they can still be displayed. Such “forced”display can be by hotkey, moving or pausing a pointer over an annotateditem, etc. The forced display can display the annotation marker or aportion of the annotation, itself. For example, a small box can come upwith a portion, or a smaller-scale, version of the associated annotationin the box.

Digital whiteboarding can be used to allow text, drawn, or otherannotations to appear in real time in other users' environments. Suchreal-time shared annotations can be stored in the manner as discussed,above, to make permanent annotations, or can be transient so that theyare not permanently stored.

Annotations can be managed by a designated manager, or supervisor, whois given special control. Annotations that are tasks to be completed byothers can be listed in a display to show when the task was assigned,whether the task has been completed, etc. The manager can close theproject for annotations. Closing the project for annotations preventsany further annotations from the users. Automatic alerting aboutcreation, completion, deletion, etc., of annotations can be provided.Pop-up windows, chat boxes, etc., can be used to report on types ofannotations, especially tasks. Such reporting can be turned off, orfiltered, as desired. For example, a task, in the form of an annotationon a clip, can be targeted to a team of graphic artists by clicking aselector button when the annotation has been created. The annotation ispropagated to the predefined team. A pop-up window alerts the teammembers that a new task has been scheduled. The header, or title, of thetask appears in the pop-up window. Clicking on the header has the sameeffect as opening the annotation so that members can learn what the taskrequires.

Reports can be generated of all of the present, and past, annotationsfor a production. Ideally, annotation information is kept in arelational database (or other flexible database) so the information canbe searched, queried and used in documents, spreadsheets, etc. Forexample, an Access database, SQL-compliant database, etc., can beemployed. Thus, annotations generated for a specific-named scene, withina specified interval of time, etc., can easily be retrieved.Annotations, and documents or reports generated from the annotations,can be made available over the Internet. This allows people workingremotely to have a high level of awareness of the different types ofwork going on in a production.

Reporting can include itemized charges of the work performed so far. Forexample, some production system operations, such as rendering, mayrequire leased, or licensed, computer time. In such cases, the costincurred for a specific rendering operation, during a period of time(e.g., daily, monthly), per user, etc., can be tracked and reported.Other processing, such as effects, transitions, etc., may require payingfor processing time and can have an indication of the cost to performthe operation.

Reporting can also include future estimates of types of work to beperformed. A keyword system can detect annotation words such as “renderthis clip” and can then determine an estimated cost for the renderingoperation based on the size of the clip. Where an operation involvesuser participation, the system can calculate the time (and, hence, thecost) of the operation based on the user's past statistics to performthe operation. For example, if a user has performed 20 color balancingoperations operations at an average of 45 minutes of user operation and3 minutes of processor time then another request for color balancing canuse these statistics to make a projection. A series of such prospectiveestimate can be used to create budget predictions for cost and time tocomplete larger parts of the production such as scenes, or the remainingwork to be performed on a movie.

In general, any data obtained from, imported from, etc., anotherapplication program can be treated as annotation data. For example, anedit decision list (EDL) from a non-linear editing system typicallyincludes scene names. These names can be treated as annotations in theproduction system. Another standardized information format is open mediaformat (OMF).

The present invention provides features for version control, storingproduction sessions and for transferring and sharing productioninformation.

As mentioned above, a production system is a large and versatile systemfor creating and handling various media. The production system includesdifferent panels, or views, of information. Each panel typicallyprovides numerous different tools, menus, displays, etc. As a userperforms operations, various windows, images, and other items may appearin the display, be moved in position and size, etc. Thus, at any giveninstant in the use of a production system there may be many variationsin the display, settings, selected tools, functions, parameters, andother characteristics of the user interface.

The changing user interface is sometimes referred to as the user's“environment.” Since environments can change rapidly, it is difficultfor users in two different places, operating two different productionsystems (or different instances of a same production system) to be inexactly the same environment. However, differences in the environmentcan cause communication problems. For example, if a first user tells(e.g., by a telephone conversation) a second user to select a menuoption, the first user is assuming that the specific menu is availablein the second user's environment. However, unless the users haveperformed the same steps in the same sequence, the environments maydiffer greatly and the menu option may not be readily available (i.e.,visible) to the second user.

The present invention allows a user of the production system to preservea “snapshot” of the environment. A user can merely activate a buttoncontrol to record all of the settings of the display at the time thebutton control is pressed. The recorded environment can be handled likea file. This allows the recorded environment to be stored, copied,emailed, etc. A second user can activate the recorded environment tohave the production system open with all of the same settings, mediacontent, etc. to recreate the first user's environment at the time ofrecording.

A preferred embodiment of the invention also allows a user to open arecorded environment where the second user does not have an instance ofthe production system software resident locally, or otherwise available,on the second user's computer system.

FIG. 2 illustrates the creation, transfer, and opening of a recordedenvironment.

In FIG. 2, User1 operates computer system 200 that includes an instanceof production system 202. Production system 202 is used to create bothsnapshot 204 and recorded environment 206. When User1's environment isrecorded the invention also saves an image of the screen as snapshot204. The snapshot can be scaled down in size, created at a lowerresolution, etc. The snapshot can be stored locally to User1's computer,or elsewhere as long as it is available to a desired destination user.For example, the snapshot can be stored via Internet 216 (or othernetwork such as a local-area network, campus network, etc.) to device208. Device 208 can be a storage device, server computer, etc.

User1 sends recorded environment 206 to other users, such as User2 andUser3 via Internet 216. When recording environment 206 is received byUser2, production system 214 resident on User2's computer 212 is used torecreate the recorded environment using settings, images, and any otherdata and information provided by the recorded environment file.

In the present example, User3 does not have an instance of theproduction system 202 resident on User3's computer system 210. In thiscase, a process associated with recorded environment 206 obtainssnapshot 204 from computer 208. The process can be part of thetransferred environment, in a manner similar to a self-extracting file.The process can also be part of a stand-alone application program, aplug-in to, or integrated with, a web browser, etc. The process may beprevented from displaying the recorded environment for various reasons.For example, the application program might be missing from User2'scomputer system, or the application program may be present but notlaunchable due to lack of system resources, lack of a needed plug-in orother process, because User2 has not given permission for automaticallylaunching the application program, etc.

Thus, even user's who do not have the necessary application program tofully recreate a recorded environment can, at least, view the screenimage of the environment. In many cases of production such viewing maybe enough to adequately communicate desired ideas and information.

The present invention provides for automated recording of anenvironment, and for automatic capturing of information at regularintervals (or based on other events). For example, a supervisor canspecify that an environment be recorded and stored whenever a specifictype of operation is performed (e.g., rendering), at specific times ofthe day, etc. This allows others to monitor the progress, andperformance, of other members of a production team. Short of recordingthe environment, a simple snapshot of the screen can be taken andstored, uploaded, published, or transferred to achieve the same goal ofmonitoring and keeping team members up-to-date on remote users' work.

The automated recording can be customized so only selected portions ofthe production system's settings, or other information, are recorded.For example, a supervisor or user can select that only annotations arerecorded and published at predetermined times. Status of publishing canbe indicated to users so that a user is aware that a clip has just beenpublished, is pending recording or publication, etc. Other embodimentscan also use this approach for settings, environments, etc.

FIG. 3 illustrates the remotely assignable control features of thepresent invention.

In FIG. 3, User1 is a primary operator of a production system executingon computer 302. Computer 302 includes physical user interfaces such asdedicated control panel 304, keyboard 306, mouse 308 and tablet 310.Naturally, any type of user interface can be used. Other types include atouch-screen, trackball, light pen, data glove, etc.

Computer system 302 executes the production system in communication withone or more other users and computer systems. A preferred embodimentuses Internet 312 for such communication, but any communicationmechanism can be used. Computer system 302 can transfer information withUser2 at computer 316. Each computer in communication with computer 302can have varying degrees of features such as user interface devices,software, etc. As shown in FIG. 3, User2's computer 316 includesdedicated control panel 318, keyboard 320 and mouse 322.

As discussed above, and in the related applications, many features areprovided by a production system. Often controls are used to direct theplayback start, stop, play speed, replaying of sections, etc., of clipportions while other operations are repeatedly performed on the clip inorder to see the effect that the operation is having on the clip. Thevarious parameters of the operation may need to be progressively changedmany times in complex interactive ways in order to achieve the desiredresult.

When one user, such as an art director, is located remotely from anotheruser, such as a graphic artist, it becomes extremely difficult tocoordinate the complex interaction necessary to achieve the desiredresult. Even where screen-sharing is provided, one of the users, such asthe director, may have a weak understanding of the operation of theproduction system and may only be able to communicate ideas about how tochange the clip by using language that does not translate well tochanging operation parameters. For example, a director may complainabout a contrast problem with certain items, or lighting conditions, ina clip. The graphic artist must guess as to the correct filter to use,the filter parameter settings to be modified, which settings to trychanging first, etc. In fact, the director's use of the word “contrast”may be a wrong choice.

The present invention allows the graphic artist (who, in this example,is the primary user and is more experienced with the use of theproduction system) to assign control of different parameters (or otherfunctions, variables, operations, etc.) to the director. For example,where the art director is User1 and where the director is User2, the artdirector can assign control of a “luminance” parameter to a slidercontrol on the director's dedicated control panel. The graphic artistcan then tell the director to move “the slider labeled #1” up or downwhile the graphic artist controls the clip playback and otherparameters, as desired. During playback, the director's computer 316executes the production system in synchronization under control of thegraphic artist's computer 302. The director's movement of the slidercontrol in dedicated control panel 318 is sent to the graphic artist'scomputer 302 so that the clip playback can be changed in accordance withnew luminance values based on the movement of the slider.

Any type of parameter modification, or other function or control, can beassigned by the graphic artist to any of the director's user interfacedevices. For example, hotkeys can be assigned to the director'skeyboard, mouse movements and button presses can be given newfunctionality, etc. The dedicated control panel can include any mannerof controls such as pushbuttons, sliders, knobs, pen and tablet, etc.Any type of input device can be used such as musical instrument datainterchange (MIDI), type of devices.

FIG. 4 illustrates drag-and-drop assignment of user interface mechanismsat the primary user's computer to control surfaces at a remote user'scomputer.

In FIG. 4, the primary user visually assigns parameter modification (orother control) to remotely located controls by dragging and dropping theproduction system's on-screen control for the parameter onto a smallimage of the target control device. In FIG. 4, screen display 400includes flowgraph area 402 and images of remote control devices at 408.When pointer 406 is moved over node 404 in flowgraph area 400, and theuser then selects node 404 to open for control, small user interfacemechanisms such as text boxes, knobs, sliders, graphs, etc., appear toallow specific control of the node's parameters.

Parameter box 407 shows parameters in the example for node 404. Theprimary user can drag-and-drop any of the parameters Red, Green or Blueonto different images of controls for remote user, User3. Each parametercan be assigned to a different one-dimensional control such as slider418 on dedicated control surface 420, mouse direction movement for mouse414, tablet direction movement for tablet 412, etc. If keyboard 410 isselected then a box appears on User3's computer system and User3 canenter an alphanumeric value from the keyboard. Where a parameter, orother control, is a momentary indication, such as a pushbutton (e.g.,when to start a clip playback), then the “parameter” (e.g., a symbolicbutton or switch on the primary user's interface) can be dragged anddropped onto a momentary control surface such as a button on dedicatedcontrol surface 420, a single keypress on keyboard 410, one of the twomouse buttons on mouse 414, etc.

Any manner of assignment of the primary user's user interface mechanismsto a remote user's control surfaces can be accomplished. Multipleparameters can be assigned to a single control. Ranges of parametervariation can be controlled by the primary user. Two or more controlscan be assigned to a single parameter. In this case, the parameter canreceive an average value of both controls, or some other derivation of avalue based on both controls. Each of the two or more controls can beassigned to different users in different locations. Any number of remoteusers can be given control of any type of parameter changing function,or other function. This provides an extremely convenient way for teamsof users to work within the framework of a complex application program,such as a production system. Other variations are possible.

A remote user need not be running an instance of the production systemon the remote user's local system. In such a case, the primary user'scomputer system can execute operations in the production system (on theprimary user's computer) and the results of such operations (e.g., asbitmap frames, etc.) can be transferred to the remote user's computerfor display.

Although the invention has been described with reference to specificembodiments thereof, these embodiments are merely illustrative, and notrestrictive, of the invention. Many deviations from the specificembodiments are possible without departing from the scope of theinvention. For example, any of the computers in FIG. 3 can act as acontrolling computer. Many different ways to achieve synchronization,data sharing and control are possible and are within the scope of thepresent invention. For example, control signals generated at computer316 can be used to directly control images on computer 316 instead ofhaving the control signal information sent to computer 302 and havingcomputer 302 generate images for transfer to computer 316. Computer 302can generate directions for playback of a clip on computer 316, asopposed to sending frames of images across Internet 312. For example,computer 302 can direct computer 316 to begin playback of a clip that isalready stored on computer 316. The two computers can remain insynchronization during the playback by methods that are known in theart.

In general, specific operations among computers or other processingdevices in communication can be performed on any of the devices andcommunicated, as needed, to the other devices.

Although the invention has been discussed primarily with respect toproduction systems, it should be apparent that many aspects of theinvention can be used with any type of application program, or variouscombinations of software and hardware systems.

Thus, the scope of the invention is to be determined solely by theappended claims.

1.-13. (canceled)
 14. A method for annotating a part of a video, whereinthe video includes frames having objects, the method comprising:determining a frame in the video; accepting a signal from a user inputdevice to define a visual annotation at a designated position within theframe to associate the visual annotation with at least a portion of anobject in the frame; and displaying the visual annotation in the frameat the designated position during playback of the video so that a viewerassociates the visual annotation with the at least a portion of theobject in the frame.
 15. The method of claim 14, further comprising:displaying an annotation marker during playback to indicate that one ormore frames in the video include a visual annotation.
 16. The method ofclaim 14, wherein the visual annotation includes text.
 17. The method ofclaim 14, wherein the visual annotation includes a symbol.
 18. Themethod of claim 14, wherein the visual annotation includes a drawing.19. The method of claim 14, wherein the visual annotation includes aphotograph.
 20. The method of claim 14, wherein the visual annotationincludes a bitmap.
 21. The method of claim 14, wherein the visualannotation includes a digital image.
 22. The method of claim 14, whereinthe visual annotation includes a thumbnail image.
 23. The method ofclaim 14, wherein the visual annotation is associated with a link. 24.The method of claim 14, further comprising: launching a web browser todisplay a web page when the visual annotation is selected.
 25. Themethod of claim 14, further comprising: launching an application whenthe visual annotation is selected.
 26. The method of claim 14, furthercomprising: playing a clip when the visual annotation is selected. 27.The method of claim 14, further comprising: opening a project when thevisual annotation is selected.
 28. The method of claim 14, wherein thevisual annotation is automatically created.
 29. The method of claim 14,wherein frames are automatically added to the video.
 30. The method ofclaim 14, further comprising: accepting a signal from a user inputdevice to select an object in the frame; and associating the visualannotation with the object.
 31. The method of claim 14, wherein thevisual annotation is stored together with other visual annotationsassociated with the object.
 32. A tangible storage medium including oneor more instructions executable by one or more processors for annotatinga part of a video, wherein the video includes frames having objects, theone or more instructions comprising: determining a frame in the video;accepting a signal from a user input device to define a visualannotation at a designated position within the frame to associate thevisual annotation with at least a portion of an object in the frame; anddisplaying the visual annotation in the frame at the designated positionduring playback of the video so that a viewer associates the visualannotation with the at least a portion of the object in the frame.
 33. Asystem for annotating a part of a video, wherein the video includesframes having objects, the system including: one or more processors; atleast one tangible storage device including one or more instructionsexecutable by the one or more processors for: determining a frame in thevideo; accepting a signal from a user input device to define a visualannotation at a designated position within the frame to associate thevisual annotation with at least a portion of an object in the frame; anddisplaying the visual annotation in the frame at the designated positionduring playback of the video so that a viewer associates the visualannotation with the at least a portion of the object in the frame.